Machine learning has undoubtedly been hugely successful in driving progress in AI, but it implicitly brings with it the train-test evaluation paradigm. This standard evaluation only encourages behavior that is good on average; it does not ensure robustness as demonstrated by adversarial examples, and it breaks down for tasks such as dialogue that are interactive or do not have a correct answer. In this talk, I will describe alternative evaluation paradigms with a focus on natural language understanding tasks, and discuss ramifications for guiding progress in AI in meaningful directions. Percy Liang is an Assistant Professor of Computer Science at Stanford University (B.S. from MIT, 2004; Ph.D. from UC Berkeley, 2011). His research spans machine learning and natural language processing, with the goal of developing trustworthy agents that can communicate effectively with people and improve over time through interaction.

Machine learning, a popular subfield of artificial intelligence that is revolutionizing everything from legal research to medical diagnostics, depends on three major parts: a model, a dataset, and the hardware that it's backed by. So how do researchers, startups and companies evaluate its overall effectiveness? Options were limited until the recent formation of MLPerf, a consortium of industry and academic partners including Google, Intel, Baidu, Harvard University, Stanford University and the University of Toronto, who are working together to offer a new benchmark suite to evaluate machine learning (ML) performance, from speed to system cost and power efficiency. "Current benchmark suites give some basic numbers to say how well these benchmarks perform on certain hardware, but do not provide any insight into why these applications perform one way or another," says Gennady Pekhimenko, an assistant professor in the department of computer and mathematical sciences at U of T Scarborough and the tri-campus graduate department of computer science. "To know which design decision is bad or not for ML applications, you want to have some representative reference model," he says.